{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hMMdZ1vnxj8f"
   },
   "source": [
    "<center>\n",
    "    <p style=\"text-align:center\">\n",
    "        <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg\" width=\"200\"/>\n",
    "        <br>\n",
    "        <a href=\"https://arize.com/docs/phoenix/\">Docs</a>\n",
    "        |\n",
    "        <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
    "        |\n",
    "        <a href=\"https://arize-ai.slack.com/join/shared_invite/zt-2w57bhem8-hq24MB6u7yE_ZF_ilOYSBw#/shared-invite/email\">Community</a>\n",
    "    </p>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "se3M7GECJfpZ"
   },
   "source": [
    "# AutoGen Agents: Evaluator-Optimizer\n",
    "\n",
    "In this tutorial, we'll explore the **Evaluator-Optimizer agent pattern** using AutoGen [GroupChats](https://microsoft.github.io/autogen/dev//user-guide/core-user-guide/design-patterns/group-chat.html) and demonstrate how to trace the process using Phoenix.\n",
    "\n",
    "The Evaluator-Optimizer pattern employs a loop where one agent acts as a generator, creating an initial output (like text or code), while a second agent serves as an evaluator, providing critical feedback against criteria. This feedback guides the generator through successive revisions, enabling iterative refinement and significant quality improvement for complex tasks. This approach trades increased interactions for a more polished & accurate final result.\n",
    "\n",
    "AutoGen's `GroupChat` architecture is good for implementing this pattern becuase it can manage the conversational turns between the generator and evaluator agents. The `GroupChatManager` facilitates the dialogue, allowing the agents to exchange the evolving outputs and feedback.\n",
    "\n",
    "With Phoenix tracing, we can gain full visibility into each refinement cycle, tracking the feedback provided, the revisions made, and the overall progress towards the desired quality standard, which aids in debugging and analysis.\n",
    "\n",
    "By the end of this tutorial, you’ll learn how to:\n",
    "\n",
    "- Set up generator and evaluator AutoGen agents with specific roles and criteria.\n",
    "- Implement the Evaluator-Optimizer pattern within an AutoGen GroupChat.\n",
    "- Manage the iterative refinement loop between agents.\n",
    "- Trace and visualize the iterative feedback and revision process using Phoenix.\n",
    "\n",
    "⚠️ You'll need an OpenAI Key for this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G2j6_RvFJfpa"
   },
   "source": [
    "## Set up Keys and Dependencies\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -qqq arize-phoenix arize-phoenix-otel openinference-instrumentation-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -qqq pyautogen==0.9 autogen-agentchat~=0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "import autogen\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass(\"🔑 Enter your OpenAI API key: \")\n",
    "\n",
    "if \"PHOENIX_API_KEY\" not in os.environ:\n",
    "    os.environ[\"PHOENIX_API_KEY\"] = getpass(\"🔑 Enter your Phoenix API key: \")\n",
    "\n",
    "if \"PHOENIX_COLLECTOR_ENDPOINT\" not in os.environ:\n",
    "    os.environ[\"PHOENIX_COLLECTOR_ENDPOINT\"] = getpass(\"🔑 Enter your Phoenix Collector Endpoint\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ibmucjY_QkaH"
   },
   "source": [
    "## Configure Tracing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.otel import register\n",
    "\n",
    "tracer_provider = register(\n",
    "    project_name=\"autogen-agents\",\n",
    "    auto_instrument=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kTVRWf0NJfpc"
   },
   "source": [
    "## Example Evaluator-Optimizer Task: Code Generation & Evaluation\n",
    "\n",
    "This **Evaluator-Optimizer** Agent pattern uses a loop where one AI agent generates an output, like code, and another agent evaluates it against specific criteria to drive improvement.\n",
    "\n",
    "In this specific example, we'll utilize a `Code_Generator` agent to write Python code based on requirements. A dedicated `Code_Reviewer` agent will then assess this code for correctness, style, and documentation, providing targeted feedback. Through this iterative cycle of generation and critique within an AutoGen `GroupChat`, we'll demonstrate how to produce higher-quality, reviewed code that meets defined standards.\n",
    "\n",
    "![Diagram](https://storage.googleapis.com/arize-phoenix-assets/assets/images/autogen_evaluate_optimizer_diagram.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qq5TvGKzO685"
   },
   "source": [
    "## Define Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dpjxM6BGLGW0"
   },
   "source": [
    "The `llm_config` specifies the configuration used for each `AssistantAgent`.\n",
    "We use a different model for code generation and evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config_list_gen = {\n",
    "    \"model\": \"gpt-4o\",\n",
    "    \"api_key\": os.environ[\"OPENAI_API_KEY\"],\n",
    "}\n",
    "\n",
    "config_list_eval = {\n",
    "    \"model\": \"gpt-4.1-mini\",\n",
    "    \"api_key\": os.environ[\"OPENAI_API_KEY\"],\n",
    "}\n",
    "llm_config_gen = {\n",
    "    \"config_list\": config_list_gen,\n",
    "    \"temperature\": 0.5,\n",
    "}  # Temperature allows for flexibility\n",
    "llm_config_eval = {\"config_list\": config_list_eval, \"temperature\": 0.5}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YWE1kjr3ZoRP"
   },
   "source": [
    "This section initializes two AutoGen `AssisstantAgents` designed for an iterative code generation and review workflow.\n",
    "\n",
    "The `Code_Generator` agent is responsible for writing and revising Python code based on instructions and feedback, while the `Code_Reviewer` agent evaluates the generated code against detailed criteria like correctness, style, and readability, providing specific feedback or a precise termination signal: `\"TERMINATE!\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coder Generator Agent\n",
    "coder = autogen.AssistantAgent(\n",
    "    name=\"Code_Generator\",\n",
    "    system_message=\"\"\"You are an expert Python programmer. Your goal is to write correct, efficient, and clean Python code based on user requests.\n",
    "    You will either receive requests for code or you will get a request for feedback from a Code Reviewer.\n",
    "    When writing code:\n",
    "    - Always enclose the complete Python code block within ```python ... ``` tags.\n",
    "    - Ensure the code directly addresses the request.\n",
    "    - Include necessary imports.\n",
    "    - Add docstrings and comments where appropriate.\n",
    "\n",
    "    When you receive feedback from the Code Reviewer:\n",
    "    - Carefully analyze each point of feedback.\n",
    "    - Rewrite the code block incorporating the suggested changes precisely.\n",
    "    - Output only the complete, revised Python code block. Do not add text or explanations\"\"\",\n",
    "    llm_config=llm_config_gen,\n",
    ")\n",
    "\n",
    "# Code Evaluator Agent\n",
    "reviewer = autogen.AssistantAgent(\n",
    "    name=\"Code_Reviewer\",\n",
    "    system_message=\"\"\"\n",
    "    You are an expert code reviewer specializing in Python. Your task is to evaluate Python code written by the Coder agent based on the original request and the following criteria:\n",
    "    1.  Correctness: Does the code seem logically correct and likely to fulfill the request's requirements?\n",
    "    2.  Compliance: Does the code adhere to standard Python style guidelines (e.g., naming conventions, indentation, line length)?\n",
    "    3.  Docstrings and Comments: Is there a clear docstring explaining the function/class purpose, arguments, and returns? Are comments used effectively where needed?\n",
    "    4.  Readability & Maintainability: Is the code easy to understand? Are variable names meaningful? Is the logic straightforward?\n",
    "    5.  Efficiency: Are there obvious major performance issues or highly inefficient patterns for typical use cases?\n",
    "    6.  Error Handling: Does the code consider potential errors or edge cases (if relevant to the request)?\n",
    "\n",
    "    Review the code thoroughly. Provide specific, constructive, numbered points of feedback referencing line numbers if possible. Focus on actionable improvements required to meet the criteria.\n",
    "    If the code meets all criteria and requires no further changes, respond ONLY with the exact phrase: TERMINATE!\n",
    "    Do NOT provide conversational text or summaries if you are approving the code with the termination phrase.\"\"\",\n",
    "    llm_config=llm_config_eval,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gWnqO5XbbAbD"
   },
   "source": [
    "Next, we define a function,`check_reviewer_approval`, to specifically detect if a message contains only `\"TERMINATE!\"`.\n",
    "\n",
    "Then, we initialize the AutoGen `UserProxyAgent` to act as the human user. We use `check_reviewer_approval` as the termination message check for this Agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_reviewer_approval(message_dict):\n",
    "    content = message_dict.get(\"content\")\n",
    "    if isinstance(content, str) and content.strip() == \"TERMINATE!\":\n",
    "        return True\n",
    "    return False\n",
    "\n",
    "\n",
    "user_proxy = autogen.UserProxyAgent(\n",
    "    name=\"User_Proxy\",\n",
    "    system_message=\"A human user providing the initial coding request and receiving the final result. Executes termination check.\",\n",
    "    human_input_mode=\"NEVER\",\n",
    "    is_termination_msg=check_reviewer_approval,\n",
    "    code_execution_config=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NzHNV2zzcTrM"
   },
   "source": [
    "Finally, we initialize the agent framework by first defining an AutoGen `GroupChat` that includes the user proxy, coder, and reviewer agents, limiting the interaction to a maximum of 10 rounds.\n",
    "\n",
    "It then creates a `GroupChatManager` to orchestrate the conversation within this group, assigning it an LLM configuration and making it aware of the  termination condition by passing in the `check_reviewer_approval` function. The `GroupChatManager` uses existing context to determine which agent to call next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "groupchat = autogen.GroupChat(agents=[user_proxy, coder, reviewer], messages=[], max_round=10)\n",
    "\n",
    "manager = autogen.GroupChatManager(\n",
    "    groupchat=groupchat, llm_config=llm_config_gen, is_termination_msg=check_reviewer_approval\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Ed8ui7-_fW2"
   },
   "source": [
    "## Run Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z3iQWLUYcrmO"
   },
   "source": [
    "We are now ready to run our agent using this sample `coding_task`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coding_task = \"\"\"\n",
    "Please write a Python function called `calculate_fibonacci(n)` that calculates the nth Fibonacci number.\n",
    "\n",
    "Requirements:\n",
    "1. The function should accept a non-negative integer `n`.\n",
    "2. Function should return the nth Fibonacci number.\n",
    "3. Include a clear docstring explaining the function, parameter, and what it returns.\n",
    "4. Handle the base cases correctly.\n",
    "5. Use an iterative approach for efficiency.\n",
    "\"\"\"\n",
    "\n",
    "print(\"--- Starting Code Generation Task ---\")\n",
    "print(f\"Request: {coding_task}\\n\")\n",
    "\n",
    "tracer = tracer_provider.get_tracer(__name__)\n",
    "with tracer.start_as_current_span(\n",
    "    \"CodeGenEval\",\n",
    "    openinference_span_kind=\"agent\",\n",
    ") as agent_span:\n",
    "    # Initiate the chat\n",
    "    user_proxy.initiate_chat(\n",
    "        manager,\n",
    "        message=coding_task,\n",
    "    )\n",
    "\n",
    "print(\"--- Code Generation Task Finished ---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pwbMnN9EfGWF"
   },
   "source": [
    "## View Results in Phoenix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ab-KW0orgVJR"
   },
   "source": [
    "Phoenix tracing shows us the Evaluator-Optimizer workflow by visualizing the iterative loop between the `Code_Generator` and `Code_Reviewer` agents within the `GroupChat`.\n",
    "\n",
    "Within the trace, you can see the entire refinement process, inspecting the specific prompts fed to the agents at each step and seeing the output progressively change and improve prior to the termination decision.\n",
    "\n",
    "\n",
    "Run the cell below to see the full tracing results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"800\" height=\"600\" controls>\n",
    "  <source src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/videos/autogen_eval_optimizer_results.mp4\" type=\"video/mp4\">\n",
    "  Your browser does not support the video tag.\n",
    "</video>\n",
    "\"\"\")"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
